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ABSTRACT 

A study compared job- performance-aided (JPA) training 
developed for a nontechnical area (census enumerawor/inter viewer > 
tfit:h training developed using the instructional systems aesign (ISO) 
model* (JPA training, an elaboration of the ISD approach, ceguiras 
redesign of job reference materials into job- perforfflance aias to 
simplify training and job performance* ) Three pairs of census ■ 
district offices, matched on variables related to difficulty of 
enuaeratioii, were randomly assigned to the two training methods and 
used training guides prepared using the two modej,s^ Data from 1197 
JPA^traiBed and 1389 control enumerators were available for analysis, 
Attitadiflal data^ collected before and after job experience, served 
as basis fior coroparison* {Only some collected performance data ware 
analyzed for reasons described Both training = approaches wer^e vary 
successful in terms of positive ratings by trainees, JPA trainees, 
however, gave significantly higher ratings of training quality , job 
preparedness^, reading ease of training materials, adequacy of 
traii#ing length, and ratings of specific activities. They also 
reported significantly higher level of on-the-job manual' use and 
better coverage of critical job tasks during training^ Supervisors 
familiar with both training approaches expressed preference for the 
JPA training* (Questionnaires, job performance record, and data 
tables are appended*) (iLB) 
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ABSTRACT 

The purpose of this study was to determine the incremental bi'^nefits resulting 
from the use of job-perfonnance-aided (JPA) instruction when compared with 
training developed using the instructional systems design (ISD) mcdel . 
Training guides for a non-techn1ca1 job area, i,e„, census enumerator (interviewer), 
were prepared using the ISD model and the JPA model. Job-performance-aided 
training is an elaboration of the ISD approach which requires redesigning 
Job reffiirence materials into jcb performance aids as a means of simplifying 
training and job performance. 

Three pairs of census district offices were matched on variables related to 
the difficulty of enumeration. The offices in each pair were then randomly 
assigned tn one of the two training methods resulting 1n three JPA offices 
and three control offices. Data from 1,197 JPA-trained enumerators and 
1,389 control enumerators wf=re available for analysis. Enumerators were 
responsible for obtaining census fnformation froni households, which failed to 
mail back census questionnaires. Attitudinal data, collected before and 
after job experience, served as the basis for comparison. Performance data 
were also collected, but for reasons described only a few of them could be 
employed in the analysis. 

Although both training approaches were very successful in terms of positive 
ratings given by trainees, the JPA approach was consistently better on a 
variety of measures. Sped ffcal ly , JPA-trained enumerators gave significantly 
higher fatings of training quality, job preparedness (after job experience), 
reading ease" of training materials, adequacy of training length, and ratings 
of specific training activities. 'Moreover, JPA enumerators reported a signi- 
ficantly higher level of on-the-job manual use (job aid manual vs standard 
manual)" and, after job experience, better coverage of critical job tasks 
during training. Other analyses indicated that enumerators with prior 
experience working in the 1980 Census reacted most positively of all groups 
studied on questions relating to training qua! ity and preparedness. Finany, 
supervisors familiar with both training approaches expressed a decided preference 
for the JPA training. These findings necessarily apply only to the six ^.^^ 
district offices involved in this experiment^ 
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^Job reference materials, such as manuals that help an individual perform 
""job tasks, have been an important part of. the work place for a long time. 
However, until the advent of programed Instruction and the Introduction of 
niodels for the systematic desujn of instruction, relatively little attention 
was paid to the design features of technical reference materials and their 
relationship to the psychol ogi cal characteristics of the user. This situation 
changed when psychoiogists, primarily in support of military technical 
training:^prograrns, initiated efforts to redesign technical manuals ta/ = 
reflect principles of human learning and to turn technical manuals j^nto 
job performance aids (Foley, 1 973). The lonq^range goals of these .efforts 
were to simplify training and the joband to produce technical materials that 
couid be used by personnel with lower aptitudes or weaker educational 
Skills. (Chalupsky and Kopf, 1967), 

The purpose of this study was to explore the use of job performance aids 
(JPA's) m a non-^technical job area outside of military technical traininq. 
Specifically, in the context of a census interviewer's Job the use of 
job-Derformance-aided trainfnn was comparsd with systematically designed 
training using standard reference materials. 

Briefly, a JPA can be defined as any information-storage device or memory ■ 
aid, such as a manual or checklist, that helps a person perform a job 
better. The present study focuses on the use of paper-and-penci 1 JPAs; 
that is, step-by-^step descriptions of a job task that are presented in^ ' 
a variety of standardized formats (e.g., checklists, and algorithms). 

As Chenzoff (1973) has emphas ized , the distinguishing feature of JPA's lies 
not in their format, but rather in the systematic, task-centered approach 
used to develop them. JPA*s are based on a task analysis and require a careful 
specification of the following characteristics of each job task: critical 
task cues, required materials or tools, decision points or questions 
related to performance of the task, and sequences of behavlDrs resulting , 
from job decisions. In addition, the development of a JPA also requires 
that training requirements, as well as the experience and abilities of 
the user population, be considered. 

The development of JPA*s, hDwever, is only one step in ,a systematic effort 
to Improve training and job performance. Ideally, the design of job 
performance aids and training should be a coordinated;Tntegrated process. 
After a task analysis, the decision must be made to include a specific 
job task either solely in training, solely 1n the job manual, or in both 
places. Obviously, the more content that can be excluded from training, 
the shorter and more simDlified training could be. ^ 

Joyce et al.{1973) list a variety of factors that need to be considered 
before making the trainlnq-manual tradeoff for a particular task. For 
example, the list (not inclusive) includes ease of learning, ease of 
communication by book, task criticality, task difficulty, reaction time, 
task frequency, environmental considerations, safety considerations, 
consequences of errors^ and the number of Individuals who perform the 
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task. Liniberry and Bullock (1980) present some simplified , decision 
rules for deciding if a task should be placed in training or not, but 
the essential rules are surnriarized as follows by Joyce et al* (1973). 
Specifically,' tasks that go into training should be. difficult to learn on 
the Job, hard to communicate with words, require a great deal of practice, 
allow little room for error'or result In serious consequences If errors 
are made, not require exorbitant sums of money to train, be performed 
frequently on the job, not allow time to refer to a manual, and be performed 
by a large number of Individuals In the job area. 

On the other hand, suitable tasks for a job aid manual are those that have 
long and complex behavior sequences^ are rarely performed, involve readings 
and tolerances, can be mentally rehearsed before the need to perform them 
arises, benefft from the presence of 11 lustrations ,* requl re reference 
Information (e*g;, tables, graphs, flow charts, schematics), and Involve 
branch-like network of possible actions based on^ existing contlngencle 
^ or job conditions. Despite the existence of these guidelines, a decis 
•td include or exclude a task from a manual ultimately depends on the 
judgement of the training designer. 

The development of job-performance-aided training follows easily from the 
developmeht of the job performance aids and their incorporation into a job 
aid manual . Since Job aids are based on a comprehensive task analysis, 
individual job aids mirror the performance of actual job tasks and can 
therefore be used to guide trainees through either Individualized or 
group learning activities. The development of job aids also removes, 
prlur to training, many of the ambiguities in procedures or Instructions 
that might exist in traditional manuals. Accordingly, instruction is ' 
simplified both for the trainer and the trainee. 

The use of Job aids in training also has other benefits. A common problem 
with traditional manuals is that the performance of a particular task ^ 
might require the user to access information in two or three different 
chapters of a manual and then, relying on memory, integrate this information 
to complete the task. In a job aid manual, on the other hand, all the 
Information for a task should be in one location, 1f appropriate, with 
accompanying illustrations. Requi ring trainees to use Job aids during 
training also encourages .act i ve, skill-based' learning, since problems 
"^caTT be p ored~ t ha t'requt r e'^t he use of^ob^a td r~for res o Tut ion .—^ - 

s) - ^ - ■ 

In summary, a job aid niani/al might have the following characteristics 

when compared with a traditional manual desianed for, the same content 

area: its organization is problem or task oriented (f.e., job conditions 

-jr stimuli are used to structure the manual rather than descrfpt^ons of 

rhe tools or equipment used in the job), there Is an Increased use of 

v1 suaT 1 1 1 ustratlons, cross-referencing 1s minimized, decision points and 

critical information are Identified, standardized formats are used to 

present task Information, a- cluttered look is avoided, principles of clear 

writing are employed (jargon Is avoided whenever possible), small performance 

steps are identified, some information and Illustrations are dupl icated, 

the manual's fable of contents Is des.1gned to encourage accessibility, 

and the education or experience of the user Is taken Into consideration. 
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The present study, investigated the use 
i,e,i census enumerator or Interviewer, 
benefits that migh^ accrue from the use 
resulting from instruction designed according to principles of instructional 
system design. Typically, in organizations with large training functions^ 
procedural Job manuals are produced independently of training materials. 
The use of a Job-performance-aided (JPA) model for training design, however, 
requires that manual and training design be accomplished concurrently* 

Accordingly, the primary purpose of this study was to compare the relative 
effectiveness of .two traimng approaches; one developed using only the^ 
instructional systems design model (i.e., training and manual developed 
independently), and the other developed using the JPA model (i.e., training 
and manual developed concurrently)* 

Variables of primary interest included attitudes about the quality of training, 
self-ratings of a trainee's degree of jrb preparedness, reported use of job 
manuals, utility of different training activities, training length, ease 
of comprehension of training materials, and potential job problems. Job 
performance variables included attrition rates, production and quality 
control measures, supervisor ratings, and scores on a final review test. 

Based on previous research, it was hypothesized that the JPA training woul d 
be simpler, more skill-based, and would emphasize problem-solving activities. 
Specifically, the amount of lecture would be reduced compared to traditional 
training. Moreover, the JPA materials should be perceived as easier to read 
and comprehend, the perceived quality of training should be equal to or ^ ^ 
better than the control training, fewer Job problems related to poor training 
should be expected or experienced, and the adequacy of training should 
be judged better after job experience. 



Expected on-the-job benefits of the JPA approach are increased use of the 
job aid manuals, closer adherence to procedures, and higher quality work* 
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To deliver low-cost, small-group training In decentralised training sitesi 
the Bureau of the Census develops training guides that are disigned to 
be read verbatim to a group of trainees. These verbatim guides reference 
all required training materials (manuals, workbooks, audiovisual s , and 
job materials) and provide step-by-step instructions for training activitieSp 

The verbatim training guides are generally used by supervisors (who are 
inexpei'ienced trainers) to train the individuals who will be working for 
them. In this studys supervisors used the verbatim guides to train 8-12 
persons for a two-day period (about 6.5 hours of actual training time 
per day). . 

Supervisors (trainers) received four days of training one week prior to the 
enumerator training session. During their training, "Supervisors received 
exactly the same training that they would be delivering to their crews of 
enumerators. 

Training Materials 

Two separate verbatim training guides were developed; one for the JPA training, 
and one for the standard or control training. 

To insure consistency between the training guides^ the author of the JPA 
training worked closely with the authors of the control training. Accordingly, 
both training guides used similar training aids and workbooks', although 
individual activities in the workbooks differed significantly because of the 
use of different job manuals. At a minimum', both training approaches were 
based on a task analysi s derived from procedural manuals and interviews with 
experts in the job area, workbook exercises were job-oriented and geared to 
training objectives derived from the task analyses, criterion-referenced per- 
formance checks were employed throughout the , training, small-group exercises 
were user to teach interviewing skilH, audiovisuals were used to cover 
critical job concepts, and a final review test was used at the conclusion of 
each training approach. 

Although there were many commonalities in the training approaches, the use 
of a job aid manual resulted in several significant differences. 1-1 rst, the 
JPA guide required approximately 25S lessjecture time by the trainer; - 
Although this resulted 1n an Immediate time savings, the additional time 
was used in the JPA approach to Include more skill-based exercises, 
provide more thorough coverage of critical Job tasks, and include a 
remedial session at the conclusion of training for those persons failing 
to obtain a pre-speci fied criterion score on a final review exercise. 
Therefqre, the JPA and control training approaches were designed to-be 
approximately equal in length. 

Another difference between the two guides was the comparative frequency 
of inserted trainer questions designed .to encourage or test trainee 
comprehension of concepts. In the JPA guide, such questions occurred, 
on the average, every 176 words, whereas in the control guide, the frequency 
was once every 293 words. 



Finany, the J^A guide could be compared tp the control tratning juide in 
terms of the relative emp^^js placed on Indiylduallzed or group»centered 
learning activities. Seventy^ percent (701) of the JPA training activities 
were individualized versus 55% for the control approach. 

Subjects 

Subjects were 2,586 adults (over 18) from six census district offices; two 
in the Northeast and four in the Midwest. All participants were required to 
pass a selection test that covered basic riading and math skills* 

Only one type of training was used In a given district office. Accordingly^ 
there were 1,197 persons trained using the JPA guides and ls389 trained using 
the control guide. 

Procedure 

Training was scheduled to he conducted at the same time for all trainees, but 
unexpected field problems caused delays of as much as a week In certain offices* 
In rto offices, however, did the^ training exceed two work days* 

At the conclusion of training, subjects in both the JPA and control apprbaches 
. completed final 'review tests and an anonymous ^nd-of-trainlng questionnaire 
(see Appendix A)* After approximately 2-3 days of job e)<per1encey, a 
similar, but expanded attitude questionnaire (see Appendix B) was again com- 
pleted by the participants. Supervisors were instructed to obtain questionnaires 
from all enumerators, even those who were released prior to 2-3 days of job 
experience, . ^ 

Since the primary Job of a census enumerator (interviewer) in this study 
was to obtain census information from households which failed to- mall back their 
census questionnaires, this particular Job was expected to last only three-four 
weeks. In^ some offices^ however, the actual time period was closer^to two 
months. All enumerators were supposed to be paid piece-rate for each 
questionnaire completed, but field problems' resulted in each office paying 
some of their enumerators an hourly wage instead. * 

During the period of emplcyment, periodic performance and quality control 
information was collectuJ for each Interviewer (see Appendix C), The 
measures Included two production Indices; one was a gross measure of the 
number of questionnaires completed by an enumerator after three days 
,of work* The other, collected on two different occasions Cafter approxiniately 
two and four days of work), was a count of the number of questionnaires 
passing a quality control edit* 

Other performance measures included a supervisor's first review of an 
enumerator's work, which involved a review of completed questionnaires 
and administrative records, plus a telephone check of a random sample of five 
households interviewed by an enumerator. Also, a record was maintained of 
the enumerator's final job status,: e.g* , assignment completed, leftHo 
take another job, etc. / _ 
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Production and quality measures were collected only during the 2-4 days 
immediately following training, because after approximatsly three work days^ 
-each supervisor was supposed to review al 1 work up to that point and provide 
on*the-job training .to deal with an^ deficiencies. Accordingly, any per--^ 
formance differences attributable to training should have been eliminated 
after this review. * ■ ^ ■■ 

Since this study was conducted in six different geographical locations * a 
full-time, data-capture clerk was trained in each office to coordinate \ 
the studyj distribute and collect evaluation insfrumentSs and deal with 
any problems that occurred. All of these -Individuals had previous census 
experience before working in this study. 

Design ' • 

Using demographic variables obtained from the 1970 CensuSj tnree pairs 

of census district offices were matched on variables thought to be related to 

the difficulty of enumeration (e.g.* density of populationj educational 

and income characteristics, mail-hack rates in t+ie 1970 Census^ etc). 

The offices in each pair were then randomly assigned to one of- the two , 

training methods, resulting in three experimental ,(JPA) and three control 

offices. ' Only one training method was used in a given office. 

Table 1 in Appendix D presents a comparison of the matched pairs of offices on 

the selected variables. 



RESULTS 

Immediately, after the start of the mail-out mail-^back phase of the Census, 
several major problems occurred that interfered with the collection of field 
performance data for ihis experiment. A critical part of each census enumerator's 
Job required the use of a list of addresses (address register) for a geographical 
area. Each address on this, list was mailed ^a questionnaire, and If one had 
been returned, an office ch'eck-in procedure indicated so by reporting the 
status^ (questionnaire received or not) of the address in the address register. 
A copy of the address register was then used by an enumerator to indicate 
which addresses needed to be interviewed. 

Unfor ately, after enumerators had been trained and had started the household 
inter /i ewi ng, numer^ous errors we're discovered In the liddress registers. Since 
enumerators spent much of their time ipnediately foljowing training correcting \ 
these Brror^^ the performance measures collected reflected both traihing 
quality and the immeasurable effects of field problems* Further, since 
it could not be assumed that field problems were comparable in the experimental 
and control offices^ meaningful statistical comparisons were precluded and, 
consequently, are not reported in this paper. 

Field problems, besides Invalidating the performance data^ also interfered 
with other data-collection efforts and quality control checks. For example. 
Table 1 shows the percentage of evaluation forms returned from the' JPA and 
controT offices* 



TABLE 1 

Percent of Evaluatton Fyrms Returned 



Type of Data Conected "jPA 

Quant1t>^ and quality of completed 83,3% 
questionnaires (after 2 days) ' 

Quantity and quality of completed ^ 76.4 
questionnaires (after 4 days) 

Supervisor's review of work ' 78.7 
(after 2»3 days) 

Supervisor's random reinterview of _ .35,3 
households (after 3-4 days) 

Gross number of questionnaires completed 89,3 
(after 3 days of work) 

End-of-tralni ng attitude questionnaire 97,6 

Post-training (after 2-3 days of job 77,9 
experience) attitude questionnaire 

Total Number of Enumerators 1|197 



CQntrdl 
. 63.1% 

50.0 

68.0 

28.5 

82.5 



79.5 



[0.3 



1,389 



1/ 
1/ 



i/ An unknown quantity of these forms/from the control group were reportedly ^- 
^ lost irf thermal 1, Moreover^ these were the only special data-col lection forms 
designed for this study. The other evaluation measures were standard census 
administrative records* 

it is readily apparent in Table 1 that significantly fewer evaluation forms 
were collectpd in the control offices* In part, this lower response rate was 
caused by field problems since supervisors reported that normal procedures 
were ignored in an effort to correct the problems in the address registers. 
Other factors, such as lack of supervisor cooperation in the offices, undoubtedly 
also played a part* i ^ 
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A 1 1 1 1 u d i n a 1 Mea s u res 

Table, 2 shows the response of enumerators; to the question "How well prepared 
do ypu feel to go out and work as an enumerator?" for two time periods » at 
the end of training and after some job experience. 

Tabli 2. 



Enumerators' Self-Ratings of Degree of Preparedness 





End-of- 


training 


Post 


-training 


H 


a. 

N - 1,158 


b. 

H = 1 ,085 


c. 

N = 919 


d. 

N = 557 




JPA 


Control 


JPA 


Control 


Wei 1 .prepared 
Adequately prepared 
Poorly prepaned 
Not prepared 


47.8% 
51.6 
0.6 
0.0 


48.2% 
. 50.1 
1.6 
0.1 


60.8% 
38.0 
1.0 
- 0.2 


54.8% 
42.7 
2.3 
0.2 



The first conclusion that can bii drawn from this table is that both training 
approaches were obviously quite successful -in making enumerdtors feel that 
they were prepared for their job'Ss both before and after job^ experience, Also^ ' 
for both the JPA and control training^ job^experiencG actuariy increased 
the percentage of enumerators v/ho felt that they had b^n "well prepared," 

The partial and marginal association of the raptors in Table 2 were tested using 
. a logrlinear model analysis for multiway frequency tables avariable In the 
^ Biomedical Computer^^Mqrams P-Series statistical package (Brown, 1977), To do this 
.analysis^ random patterns of missing data were assumed. Further, it was assumed 
that field problems and supervislbn did not d'l ffe'r 'between JPA and control offices. 
The best-fit model Included all two-way interact lonSi but not the three-way 
interaction. The marginal association between degree .of preparedness and experi- 
mental groups (l.e^, , between columns a. and b, ) was not significant at the 
^ end-of-tratnlng comDarison (x2 * 6,15, 3 dfi p ^ 0,1), bCit it was significant • 
(x^ = 8,45, 3 df, p = 0,04) at the post-training comparison (i.e., between 
columns c. and d,). 

Since some research JPA*s has Indicated that less experienced personnel using' 
pob aids were able to perform at levels of proficiency comparable to persons 
with more job experience, it was hypothesized that enumerators who differed 
in prior census experience would not differ in their feelings at job preparedness 
after JPA training^ but they would differ after standard census training. The 
same relationship would also hold for the post-training questionnaire results. 

Although asklrfg about job- preparedness is obviously not the same thing as 
measuring actual performance, "Table 2A 1n Appendix D shows that the hypothesis 
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was not supported . For example, comparing columns a. and d. In Table ZA, 
60% of JPA enumerators with prior experience felt that they were "well prepared" 
versus 45,1% of the JPA enumerators with no prior experience* The response 
patterns in columns a. and d. differed significantly, although the identical 
comparison for the post-training questionnaire dtd not (see Table 2B, in 
Appendix D).^ Also, contrary to the hypothesis^ for the controV enumerators 
there were no statistically significant differences on either questionnaire 
wffen experience was controlled for. 

Tfie results also suggest that JPA enumerators with prior census experience 
reacted more favorably to their training than control enumerators with prior 
census experience^ but these differences were not statistically significant 
(see Table 2C, Appendix D). 

Table 3 shows the responses of enumerators to the question "Overall, how 
would you rate the quality of the training?" 

TABLE 3 



Enumerators' Ratings of Quality of Training 





End-of 


-tra ini ng 


Post- 


-training 




at 


b. 


c. 


d. . ' 




N = 1,163 


N = 1,093 


N = 887 


N = 526 




■JPA 


Control 


JPA 


Control 


Very Good 


57.0%' 


49.0% 


58.7% 


/ ' 44.9% 


Good 


37.8 


•42.1 


37.1 


46.6 


Fair 


4.S 


8.3 


3.0 


8.0 


Poor 


0.7 


0.5 


1.1 


0.6 



Again, it is readily apparent from this table that both training approaches 
were well received both immediately after trainin'g and /after job experience* ^ 
However, a log-linear analysis, identical to one' performed for the previous^ 
table, indicated that the JPA group gave /statistical ly higher ratings of, training 
quality* The marginal association between quality and experimental groups was 
significant for both the end-of-training (x^ = 22,7, 3 df , p - 0,00) and 
post-training comparisons (x2^= 36,^8, 3 df, p - 0.00)* 

Although there were no findings in the research literature which' would predict 
how more experienced personnel would rate the quality of JPA. training, it was 
hypothesized that JPA-tra1ned enumerators with prior.- census experience would 
gi^e higher ratings of quality to JPA training than JPA-trained enumerators 
without prior experience, since they would have had previous census training as 
a standard for comparison* This outcome should not occur for control enumer- ' 
ators, unless the standard (control) training in this study also deviated 
qualitatively from typical census training packages* 



Breakdowns of ratings of training quality, control ling for prior exptri^j^c^* 
are presentid In Tib],e 3A In Appendix D, But, although the differences 1n 
the hypothesized dirfectionj they are not statlsticany significant (see "fibU 
3B in Appendix D). 

Again, looking at just those groups with prior experience in Table 3A, 
proportionally more JPA-trained enumerators gave "very good" ratings to 
their' training than their control counterparts. Both of these comparisoi^g 
were statistically significant (see Table 3C in Appendix D). 

Since one objective of JPA*s was to simplify the presentation of inforfna^i^n 
and Improve comprehenslbillty, a logical question was to ask whether trapping 
materials were easy to read and understand. Table 4 presents the respon^g^ 
to this question which was asked only at the end of training* 

' TABLE 4 

Enumerators' Ratlfigs of Training Materials Com'pfehensibillty 



Yes 
No 



N = 1,150 

JPA 

95.8% 
4.2 



N = 1,080 

Control ' 

90.7% , 
9.3 



The trarning materials in both approaches were judged easy to read and ■ 
understand, but the JPA approach elicited a small, but iignificantly higKjgr 
rating (x? =.-23.4, 1 df, p^ .00). 

Table'^ presents the responses of enumerators to a questiSn about the lei^gth 
of training. 

TABLE 5 , . 

' Enumferators' Ratings of Length of Training ^. 





N = 1,162 


N = 1,088 






JPA 


Control 




loo Short 


12.1% 


15.1% '-^N 




' About Rigtit 


81.0 


• . 72.5 




. Too Long 


6.9 


12.4 





It can be seen that proportionally more enumerators rated the JPA trainir>gas 
acceptable in length (x^ = 26.8, 2 df , p = .00). 
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Of major Interist at the complitlon of any training Is the self-confidence a 
trainet feels about his orjjer ability to accomplish critical job tasks, - 
Accordingly, enumerators were asked to check off major job tasks that they 
thought would either cause them difficulties on the job (asked on end-of-tra1n1ng 
questionnaire) or which were not covered welV enough In training based on 
their job experience (asked on post-training questionnaire). 

Table 6 presents a cwnparlson of these job tasks based on the number of people who 
checked a task. This number was converted to a percentage and 1s reported in 
parentheses.' , 

TABLE '6 

Enumerator-identified "Problem" Job Tasks 



End-of-Training 



Post-Traini ng 
(Not Covered Well Enough 



t 

* 






in Training) 






N = 1,168 


N = 1,097 


N = 932 


N » 560 


TASK 


JPA . 


Control 


JPA 


Control 


Locating addresses 


8.0% 


6.9% 


8.3% 


9. 5%' 


Interviewing people 


11.6 


9.1 


5.9 


6.8 


Completing the questionnaire 


7.7* 


S.6* 


5.2* 


8.9* 


Completing the address register 


4.6 


6.0 


4.6* 


9.1* 


Fining out payroll forms 


8.2* 


14.9* 


11.3* 


15.7* 


Using the job reference manuals 


5.1* 


9.2* 


4.4* 


8. 6* 


Getting people to cooperate 


49.4* 


36.9* 


14.7* 


20.7* 


Dealing with unusual situations 


42.6 


39.1 


17.8* 


28.0* 


Answering respondent-questions 


10.4 


10.8 


6.9 


8.4 


Checking for missed housing units 


19.4 


18.6 


12.7 


15.5 


♦Indicated that chi-square statistic 


is significant. 


p < .05, 


1 df. 





A chi-square statistic (job task, checked or not checked^ by training method) 
was computed for each task at the end of training and after job experience. 
At the end of training^ only four job tasks produced significant chipsquares 
(p < ,05), These job tasks dealt with completing the questionnaire^ completing 
payroll forms, using job reference manuals, and gaining the public's cooperation 
to complete the census form« 



For two of these tasks--gaining cooperation and completing the questionnaire-" 
the JPA-trained enumerators expected more problems. However^ the JPA-tralned 
enumerators expected less problems in c6mpleting payroll forms and using 
job reference manuals, . 



After job experience, however, there were significant differences on six 
job tasks, and in all cases the control group felt that the tasks could have 
received more el^phasls during training. These six job tasks dealt with completing 
the questionnaire, completing payroll forms, using job reference manuals, 
gaining the public's cooperation, dealing with unusual situations, anw completing 
the address register. ^ 

As with previous analyses, the effect of prior census experience on an enumerator' 
self-confidence in preparing job tasks was of interest. Accordingly, It was 
hypothesized that JPA-trained enumerators, whether or not they had prior 
experience, would not differ in either their expectations of expected job 
problems or their opinions about which tasks needed more coverage in training. 
On the contrary, it was expected that control enumerators, with and without 
prior experience, would differ significantly 1n their reactions because of 
the absence of job aids to guide performance* 

Table 6A In Appendix D presents a breakdown of Table 6 controlling for prior 
experience; however, the resulrts do not support the hypotheses (sea Table 6B 
in Appendix D). Although there was only one significant difference for the 
JPA-trained enumerators on the end-of-tralnl ng questionnaire, there wis also 
only one difference for the control enumerators. Moreover," on the post- 
trajning questionnaire, there were two significant differences for the JPA 
group, but only one for the control group. 

A comparison between JPA and control enumerators for different levels of 
experience also showed (see Table 6C in Appendix D for statistical tests) 
that the penceptions of JPA and control enumerators with prior experience 
differed on only one-third the tasks that JPA and control enumerators without 
prior experience differed on both before and after job experience. 



Although it was not possible 1n this study to obtain actual observational 
data on manual use, the question "On the average, how many times each day do 
you use a particular^manual ?" was asked of all enumerators after two-three 
days of job experience. Table 7 shows the average reported dally use of 
the two primary job reference manuals, either the job aid or standard manual, 
and the Questionnaire Reference Book (QRB) which provides detailed Instructions 
for each question asked on the census form. 



TABLE 7 



Reported Average DalTy Use of Job Reference Manuals 



H ^ 899 



N 



= 532 



JPA 



Control 



Job Aid or Standard Manual - 2.02 (2.13)1/ 
ORB 1.04 (1.66) 



1.77 (U81) 
1.27 (1.59) 



i/ Standard deviations are reported in parentheses. 



An analysis-of-variance model for nested factors (census district offices 
nested under training approach) was used to deterniine if the reported use 
of the job aid manual or standard manual differed significantly in the 
experlrental offices, (Winer, 1962, p, 184), Table 8 presents the results 
of this analysis. 



TABLE 8 

Analysis-of-Variance Table for Manual Use 



Source 


df 


Sum of Squares 


Mean Squari 


F 


Training Approach 
District Offices • 
Residual 


1 . 

4 

1431 


20.2 
200.1 
5612.7 


20.2 
50.9 
3.9 


5. 1 3 i/ 
12.7 1/ 


i./ Significant p < 


.05 









This analysis showed that, not only were there significant differences in reported 
manual use between the JPA and control groups, there were also significant 
variations in manual use attributable to district offices* 



Although the use of the QRB was not of primary interest in this study, because 

it was not redesigned (i.e., the same QRB was used in both training approaches), 

1t Is Interesting to note that its reported use was significantly higher in 

the control group (F=7.17; p <*05) than in the JPA group. Also, the correlation . 

between manual use and QRB use was 0,59 In the JPA group and 0.64 in the control. 

Further, prior census experience did not affect use of^either the Job manual 

or the QRB, 

Another question on the post-training questionnaire asked each enumerator to ^ 
give a self-rating of job performance. There were no significant differences 
in the response patterns shown in Table 9. 

TABLE 9 . ■ 

Enumerators* Self-Ratings of Job Performance . , . 



N ^ 901 . N ^ 549 

JPA 



Very Good 37.4% 

Good 52.4 

Fair 9*4 

Poor 0,8 



Control 

34. 8S 
54.5 
10.0 
0.7 



As mantloned previously, JPA-deslgned training should differ quail tatlvtly 
from standard, systematically designed training. Accordingly^ enumerators 
again were asked on two different occaslans to rate the utility (most helpful, 
helpful, least helpful) of training activities In helping them learn to do 
their job. Table 10 presents the ranking of these activities derived by 
computing the average rated helpfulness of each activity where "1" was most 
helpful and "3" was least helpful* Standard deviations are reported in 
parentheses* - 

TABLE 10 



Enumerators* Mean Ratings of the Helpfulnesr^of Different Training Activities 









End-of-Tralning 




Post-Training 






JPA 




Mean 


Control 


Mean 


JPA 


Mean 


Control 




Mean 


Workbook 


1 


.28 


(0.4€) 


Prac Int 


1.46 (0.59) 


Manuals 


1.39 (0.56) 


Prac Int. 


1 


.55 (0.66) 


Manual s 


1 


.31 


(0.491 


Manuals 


1.48 (0.57) 


Workbook 


1.41 (0.54) 


Manual s 


1 


.62 (0.61) 


Liicture 


1 


.42 


(0.57) 


Workbook 


1.53 (0.60) 


Lecture 


1.49 (0.58) 


Lecture 


1 


.63 (0.61) 


Prac Int . 


1 


.56 


(0.64) 


Lecture 


1.54 (0.60) 


Prac Int 


1.69 (0.68) 


Wookbook 


1 


.68 (0.62) 


' Audio VIS 


1 


.60 


(0.67) 


Audiovis 


1.56 (0.68) 


Audiovis 


1.78 (0.73) 


Audiovi s 


1 


.?4 (0.73) 



i/ Audiovisual presentations used filmstrip and tape. End-of-tra1ning percentages 
are based on, a miriinium of 1,039 respondents; post-training percentages are based 
'on a minimum of 539. 

Practice Interviewing was viewed as generally most helpful on both occasions 
in the control group, whereas in the JPA group, its mean rating was second from 
last In both instances* 

The JPA groupi on both. occasions, ranked training activities that used either 
the job aid manual or the workbook as the most useful. Although this ranking 
differed from the control group's, a comparison of the mean ratings was not 
statistically significant* 

One problem with the preceding table Is that it uses average ratings to 
compare training activities. The use of averages could tend to obscure 
significant variations in ratings* For example. If half of a group rated 
an activity as "most helpful" and the other half rated the same activity as 
"least helpful", the average rating would be "helpful" which is not Indicative 
of the true Impact of the training activity on the trainees* Accordingly, 
Table 11 presents another ranking of the training activities on. two. 
separate occasions. However, In this case, the rankings are based on the 
percentage of enumerators in each group who selected a particular activity as 
••most helpful". 
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TABLE 11 

Enunierators' Ratings of "Most Helpfur' Training Activities 



J PA 



End-of-Tralning 
Control 



Post-Training 



Workbook 
Manual s 
Lecturt 
Prae Int 
Audiovis 



(73.5%) 

(69.7) 

(61.7) 

(52.3) 

(50.7) 



Prac Int 
Manual s 
Audio vis 
Workbook 
Lecture 



(59.7%) 

(56.0) 

(54.9) 

(52.4) 

(51.3) 



JPA 

Manuals 
Workbook 
Lecture 
Prac Int 
Audiovi s 



(63.1%) 

(62.3) 

(55.6) 

(43.4) 

(40.3) 



CQntrol 

Prac Int 
Manual s 
Lecture 
Audiovi s 
Workbook 



(54.2%) 

(45.0) 

(44,5) 

(43.4) 

(40.4) 



The rankings of training activities between Tables 10 and 11 are Identical for 
the JPA group. For the control groups however^ audiovisual^, which had the lowest 
average rating of helpfulness in Table 10^ fared better when the rankings were 
based on the number of times an activity was checked "most helpful." 

It Is also of interest to note 1n Table 11 that there was more agreement 
among the JPA enumerators about the most helpful training ad^ivities. On 
the end-of-tralning questionnaire the workbook and manuals were rated most 
helpful by 73.5% and 69.7% of the JPA enumerators, Whereas on the post-training 
questionnaire, these percentages dropped to 63.1% and 62.3%. For the 
control group, practice Interviewing and manual use were rated most helpful 
by 59.7% and 56.0% after training, with these percentages dropping to 54.2% 
and 45.0% after Job experience. 

Although the training activities in Table 11 differed in the JPAWd control 
groups (a.udiovisuals were identical), a comparison of how enumeratprs rated ^ 
the helpfulness of different generic training activities in their respective 
groups provides insights into their success in teaching job skills. In the 
development of materials for this study, It.was evident that design deficiencies 
In existing job manuals severely restricted the utility and the types of 
learping activities that could be included, for example. In the workbook. ^ 
Therefore, although the workbook in the control training was an integral part 
of the .training and contained exercises similar to those in the JPA workbook, 
ii nonetheless received far fewer ratings of "most helpful" from enumer^^tors. 

Additional Variables 

Although a final review test was part of both training approaches, the same 
test was not used. Therefore, comparisons between training methods were not 
possible. Separate, one-wayvanalyses of variances were computed, however, for^ 
each group. Table 12 presents these results whlch^are based on 84 randomly 
selected tests from each district office. These tests were selected and re-scored 
to ensure consistency In the scoring standards. 



a." Analysis of Variance for JPA Final Review Tests 
Source df Sum of Squares Mean Square £ 



I 



Betwetn 

Within 

TOTAL 

b. 

Source 

Between 

Within 

TOTAL 



2 

249 
251 



26.7 
5822.5 
5849.2 



13.36 
23.38 



.57 



,57 



Analysis of Variance for Control Final Review Tests 
df Sum of Squares Mean Square £ 



2 

249 
251 



92.4 
1352.1 
1444.5 



46.19 
5.43 



8.51 



.0003 1/ 



NOTE: The JPA final review test was criterion referenced; the control review 
test was multiple choice. 



J./ Using the Scheffe' post-test procedure, average scores In one control 
office were found to be significantly lower ( p < .05) than the other 
two. 

The significantly lower test scores found for'one of the control offices In 
table 12b could theoretically be caused by differing aptitude levels of 
trainees in the different offices. Table 13 shows the average scores obtained 
by enumerators , on the ^election test used for this Job, The control office with 
the lowest average selection scores did, in fact, perform significantly worse on 
the final review test, but the mean difference in selection scores was not 
statistically different. 

TABLE 1 3 

Average Selection Test Scores for Enumerators 1/ 



UPA Stan dev 

36.0 8.4 

37.7 8.4 

38.8 8.1 

Weighted Av 37.6 8.3 



Control 

35.1 
38.6 
38.7 

Weighted Av 37.4 



Stan dev 

8.8 
8.4 

8.4 

8, 7 



- 1 



kl Maximum score is 54. 
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Other variables that theoretically could affect the attitudinal varlabTts 
reported so. far, tspecially the post-training results, are pay rates and 
questionnaire mail-back r^tes for the experimental groups. Although pay 
rates were supposed to be based on piece work for the enumerators in this 
study, managers of different offices were given the option to put people 
on an hourly rate to deal with unusu.il field problems. Mall -back rates are 
Important becaurt, in neneral , more difficult, areas of enumeration have lower 
mail-back rates, ' 

In the JPA and control .offices, ultimately 22X and 24% of the enumerators, 
respectively, were put on an hourly pay scale. The mail-back rates for 
these groups were 84.2% and 86%, respectively* Neither one of these differences 
was large enough to affect either the conduct or conclusions of this study. 

By the end of this particular census operation 62.7$ of the enumerators 
in the JPA offices completed their assignments, compared with 55*8% in the 
control offices. Major reasons for failures to complete assignments are 
given in Table 14 as percentages of the total number of enumerators not finishing 
their assignments* / / 



TABLE 14 

Ranking of Major Reasons for Enumerator Separations 



N = 1,140 




N = 1J60 




JPA 




Control 




Pay dissatisfaction 
Work dissatisfaction 
Poor performance 
Take another job 
"othfer'' 


3.2« 
4.2 
4.9 
7.6 
17.4 


Pay dissatisfaction 
Take another Job 
Poor performance 
Wo !■ ;4 d 1 s sa t i s f a ct i 0 n 
"other" 


1.7% 
6.2 
10.7 
10.9 
14.7 



It woyld not be meaningful to calculate any statistical comparisons- between 
these groups because of the field problems mentioned earlier. For example, 
differences in attrition rates could be caused by field problems, differences 
in supervision, differences in pay systems, training differences, or a com- 
blnatldn of the preceding. There Is some evidence, for example, that fTsld 
problems were ^somewhat worse In the control offices than in the JPA offices. 
This evIdenH^ comes primarily frofn ff^ on the post-training 

questionnaire. Table 15 presents a summary of 'these comments expressed as 
a percentage of the number of people completTng the questionnaire. 
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TABLE 15 ' , 

Common Problems 1n Census Operations Reportecl by Enumerators 



'N = 932 " 




N * 560 








W Wl i Ip i W i 




Problems with address registers 
Inadequate pay 


(8.4%) 
(3.4) 


Problems with address registers 
Inadequate pay 


(6.4%) 
(4.3) 


Bad maps 

Poorly designed questionnaires 


(1.0) 
(1.0) 


Followup started too soon 
Poorly designed questionnaires 
Poor training 
Bad maps 


(3.6) 
(1.4) 
(1.0) 
(1.0) 



In this table, problems with address registers and a too early start of the 
followLip operation to obtain census information from non-respondents , probably 
refer to the same problem* For example. If the followup interviewing started 
before all questionnaires were checked In, the address registers would contain 
incorrect information. ^ 

One final extraneous Variable of interest concerns the experience levels of 
enumerators. Based on an analysis of responses to an item on the end-of-training 
and post-training questionnaires, which asked about previous job experience, 
the enumerators In the control offices were found to be^more experienced 
than their counterparts in the JPA offices* 



TABLE 16 



Enumerators Reporting Prior Experience 
Working in the 1980 Census 



x. 



End-of-T raining Post-Training 
JPA 12.5% ^ 11.5% 

Control 17,3 . 18.9 



X 



Although these differences were statlsticany significant (p < .OSMn ^ 

ToCT^^occi¥TonsT^ prior expiriince 

in both groups changed little from one questionnaire to the next. This 
finding supports the assumption that, although the number of missing 
questionnaires was higher in the control group* the sample of questionnaires 
obtained was representative, ( 



DISCUSSION 

The major objective of this stud^ was to determine the incremental benefits 
that resulted from the use of job-performance-a1ded Instruction, (i^e.* 
systematically designed Instruction that- used job aids)* when compared with 
instruction that only Incorporated principles of Instructional system design* 
Incremental benefits in this case were defined as measurable performance 
differences related to the quantity and quality of an enumerator's work, 
attitudes about training and training activities^ and reported use of job 
manuals. A secondary objective was to demonstrate that training and Job 
manuals could be simplified using within^^house training capabll Itles* 

The results Indicated smalls but consistently higher, ratings for the JPA 
training on questions about priparedness (post-training), quality 
(end-of-training and post-training), donjprehensibll i ty of materialSj length 
of training, and specific job tasks (particularly after Job experience). 
The differences between the JPA ^ j control groups in perceptions of Job 
tasks that were expected to cause problems (end-of-training) , or which were 
not covered well enough In training^ were particularly Interesting because 
they could be linked directly to differences in training content or emphasis. 
Andj after job experience^ all the differences indicated that the JPA training 
had done a better Job of coven^ng major J^pb tasks. 

Although this study was unable to compare the Job performance of JPA enumerators 
with and without prior census experience, comparisons on attltudinal items . 
indicated that JPA enumerators with prior experience gave proportionally 
higher ratings of preparedness immediately after trainingt A finding that 
did not occur for the comparable groups of control enumerators. However, 
prior -expe rience als o seemed to lessen th e effects of dif ferences in the 
training approaches/ Specifically, although JPA and control enumerators 
without prior experience differed in their expectations on three Job tasks 
that they thought would cause problems at the end of training, the JPA and 
control enumerators differed on only one. After job experience, the JPA 
and control enumerators without prior experience differed on six tasks that 
they thought were not covered well enough in training, whereas the JPA and 
control enumerators with prior experience differed on only two. 

One of the more important v^ariables In .this study, especially in the absence 
of valid performance data, was the average dally use of manuals reported by 
enumerators in the period of time Immediately following training. Since formal 
training was relatively short, the manuals served as an Important reference 
source for Job tasks. Accordingly, the more they were used, the better job 
performance was expected to be. Although both the JPA and control, enumerators 



reported using job reference materials (proctdural manual and pRBl^^j^^ 
-— threr-Mmes^Mch^day—^^ 1 maqiua 1 (job 

aid manual) more frequently (66% vs 58%). It appears* therefpre, that increased 
use of the job aid manual in tNiining translated to increased use of the 
manual on the job. Other evidence to support this conclusion came from the 
tnumerators' ratings of the adequacy of training on major job tasks, both 
prior to and after job experience. At the end of trainingi only 5.1% of the 
JPA enumerators expected that use of job reference manuals would cause problems, 
compared to 9.2% of the control group. After job experience these percentages 
dropped to 4.4f and 8.6%, respectiyely. 

\Further evidence that job-perfonriancf-alded training was qualitatively 
different from training not using JPAs is proyided in Tables 10^ and 11, 
The JPA-tra1ned enumerators consistently rated using manuals and workbooks 
as the two most helpful activities for learning the job*, whereas the control 
group consistently rated practice interviewing and using manuals as the 
two most helpful activities* In general , the JPA group also gave the training 
activities higher (more positive) ratings, again reflecting more ^ef feet Ive 
training. To reiterate, however, enumerators 1n both trainrng apprbaches' 
gave all the training activities an average rating of at least "helpful". 

Based on the preceding discussion. It appears that job-performance-aided 
training had a significant Impact on trainees' attitudes and reported use 
of procedural manuals, even though ratings of the control training were 
qu1^;e positive. As with any experimental study, however, there were several 
other variables that might have had an Impact on the results. 

first, the job»performance-aided instruction used in thJ^ field experiment 
was developed and implemented within the limitatiohs ijnposed by the use of 
a verbatim training guide, a mode of Instruction not widely used or even 
recommended by training specialists. As Swezey (1977) and Fredericks- 
Hoover-Rice (1977) demonstratedf*however, JPA's can be adapted to a variety 
of instructional approaches (e#g. , learner-controlled Instruction and 
computer-based training). Actually, it is pointless to think of usijig 
JPA's independently of modes of instruction^ sinqe the choice of Instructional 
mode should influence the'uK^ and effectiveness of JPA*s la training.' 

Also, with respect to this study, t^ie use of verbatim training guides 
' Introduced /Controls which may be dl^Pflcult to match in future studies. 
Specif leal ly^ the use of a verbatim guide, although a limiting factor 
instructional ly, nonetheless Insuredthe standardized presentation of both 
training approaches, ^ 

.^Secondj unlike most of the evaluative studies of JPA's, this study dealt 
with a non-technical area and, except for enumerators with prior census 
experience, users did not have a chance ^to compare different manuals 
(e,g., job aid vs standard). Moreover, the training was Intense but limited 
in duration (two work days). And the expected length of the job was only a 
month, although in one experimental office the actual period was closer to 
two-and-a-half months, . ' 
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Third, the tratner popuTation was widely divergent -In terms--of-educatlon . 
Mf)d experience. Although actual demographic characteristics are not yet 
avanable for enumerators, evidence from past censuses indicates that 
trainees range from the marginally literate to persons with advanced 
degreeSv Generally, howevers weak educational skills were considered to 
be the norm when both training approaches were designed. 

Fourth, as with any large-scale field studyj the size of the effort and the^ 
Inability to control major sources of variation (e.g. ^ field problems) 
force a cautious interpretation of the results. In this studyj serious 
field problems occurred in all participating offices, but It was impossible 
to determine if the effects were more severe in the JPA group as opposed 
to ^the control* Although certain findings can be used to argue that field 
problems were worse in the control offices j which could have Influenced 
post'-training attitudes, there was additional evidence that field problems 
were Just as severe in the experimental offices. For examplt, the last 
office to close out its census operations was a JPA office, and another JPA 
office discovered that approximately 201 of Its mall returns were misplaced . 
by the -U.S. Postal Service for two weeks Into what was projected to be a 
three to four week operation. The response problems. Indicated In Table 1 
for the control offices reflect, 1n part, a lack of cooperation by supervisors 
In those offices, as well as the Inability of data^capture clerks to establish 
good working relationships with the field staff so that materials were 
distributed and collected* 

Finally, all conclusions about post-training differences on evaluation items, 
or changes from end-of-tra1n1ng to post-training, must be qualified because of 
the large differences, In the percentages of evaluation forms returned from 
the JPA and control offices. These differences may reflect the failure of 
data-collection clerks in these offices to* do their jobs, or they may reflect 
situational differences that could have had a confounding effect on the 
measures used.' Contact durlag the study with the data-collection clerks 
Indicated that failure to establish good working relationships with field 
personnel, turnovier of clerks, and competing office responsibilities may have 
contributed to. a failure of clerks in the control offices to obtain as many 
forms as the JPA^ clerks. 



Cost of JPA Development 



In general, bagged on dil^cUrstons in the litirature, 1t appears that JPA's 
cost mort to produce, but as Chenzoff (1973) also noted^ there are no good 
cost comparisons bttwien JPA*s and standard -approaches for dtvelopirig 
techn^al materials. Unfortunately, this study did not directly address this 

Issue.^*--^^--^^ ' 

Part of the probltm in generating cost estimates is that two different products 
are actually being compared, since JPA' s- require a different approach to training 
design' and development. JPA's, for example, definitely require more design time* 

Other .ifDportant Considerations when discussing the costs of JPA's concern the 
selection of fonna||/Sid the job area* ^.For Instance, job aids developed for 
maintenance tasks in the military often employ photographs and complex 
schematics, both of which are expensive to produc^^ Moreover, hardware dhangeSj 
which are common and frequent, require expensive revisions. 

In cootnast, the present study used simple and Inexpensi vely^produced job 
aids which could be produced in draft form by a typist for testing and validation. 
However, final copy required the aid of a graphics- specialist to lay out the 
manual. ^Moreover, once procedures were finalized, ttfe job aid manual required 
no further revisions, . but even had they been required, changes could easily^ ^ 
have been^accomoyated. 

Although any conclusion must be qual 1 fled, it was thefeeHperlence In this study 
that any Increase in costs due to the' Increased design and development time 
required for JPA's was offset by shortened design and development time'for 
accompanying training, more effective training, and improved trainee attitudes. 
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AfPffiffilX A Sm-OF-TRAIIIIMG QDUTIQNNAiai 



I 



BlUMiRATOR TRiilNING EVALUATION 

^th Qs^eiifiid Csn%um - 19S0 



INSTRUCTIONSi Tht Suftau of ihi Cinsui 
wants ^ ioiQw hsw ydu fsti ibduC ch€ 
tmining xhut ym jusi finishtd. 

Pltasa answtP ihe quistisns below. Do ngf 



fNTfi? rouj? OlSTmcr office code numbsr herb 



U li riiU y©yr first \mb'^tkmq \n th® 1980 isfliys? 
2 O No 



2. How w«il ptmpmf^d da f«y f#«i fra tut ind wark 

as 991 9lHliQ#fSt^? 

M3m {)<) one 

f rn Well pre^ftd 

3 2 Pgof ly prepared 
« ~ Nqc prepared 



3* Wh#f® mmtm jqq trglnad? 

I Q irr a foom with 3 or mcsft pcoplt 



1 ^ On Che jab - Hew iMf Hi th© ^inirif Igst? 



Ateffe Qn(X) in liiMpim^riQtB column that btfc 
dtserfbM ^ow igcftSctiviey rte/ped you ieam 



Lectufe by Winer 
(reading from ifiinlng 
luide) ^ * 

b. Practice Intefviewjni 
(eiiher with Brminaf or 
faiiaw efiumtratars) , 



Wofkbaok extf cists .... 

4, Reading of using 

manuals — * 

Audiovisuali (slides and 
^pe racardmis) 



V«fy 



helpful 



Mark (X) one 
1 ^ T@Q shaft? 
1 Q A&#yt right? 
3 Q Taa lanf ? 



fi« Qv#failf ha^ wauiy yay rata th@ qyaiity af th# 

fX) an€ 
- ! □ Vefy |QOd 
I ^ Gasd 

3 □ Pair ^ 

4 Q pQdP 



7. Wtra tfi# ti^inmf mattrials tasy ta fMd and 
yndtfitand? 



S, Tli# fallawifif is a Hit a* dltferant parfi af yayr^^ 
jab« ' ' . 

/Viorfc on (X) by Wh thai you think w?// CQUSe 
iomt pfob/ami when yoy start workings 
t ^ Locatiirf^ddresses *at did not re^rn 
a questionnaire 

2 □ Int^viewini people (asking the questions) 

3 □ G4mpl&£ing the qyestiannaire (address label, 

'-Pof Census Use Only'* box, etc.) 

A Q Completi^ the address register 
s ^ Pilling out payfQii fom%/\^: 
8 Q Using tfieejab feferenei manuals 
7 □ Getting people to cooperate 
§ Oeanng witfi, unusual^ situations 
f Q Answifing questions asked W respondent 

10 2 Checking for fnissed housing units 

1 1 ^ Other - Specify > / 



9, What one tyng did yaw disiike mait^fib'sut tht 
trgininf? ^ 



If im^^r^i 



TRAINING AND PfRFORMANCS EVALUATION 
FOLLOWUP iNUMiRATOR 

2Qth D««Mniai C®fuus - 1980 



I J.' - 



h INSTSUCTlQNSi Befsre fQu Ngan w^ifig. you «m* 
■ pitted a qyiStienniirt absui your rninini hr *il job, 
Nqw yauhavi had same jgb enpefieiiM* Surtau 
§f thn Census wanes m knew haw tfiac winini halpad 
yeu in yaur Jeb, ^d haw ysu fial abeyt yguf job* 

Plaase inswtf th© gyeitians balsw* D© Mt write yaur 
art this fann^ 



ffNTffif yOUff 0/STR/CT O^FICI CODf HUfABER HERS 



U U Aii y««ir ft«t i«h w«fkinf m Hut IfM €«fiiy»? 
! ^ Tts 



an OTumof^tar? 
MsHi (X) ana 

I ^ Wal I pf eparid 

I Q AdMuamiy prapafad 

1 Paafiy prepaid 

« ^ Nac pr©paf#d 



3, Whaf9 w«f« yau trained? 

t 2 in 1 faam widi 3 af mara p^apla 
2 On tha jab 



4^ Thm fsl lowing is a lUt af diH«rwf ^fti af yayr 
job*. 

/mrk art fX) by eedi that was net isvimrf^ 
wai/,anoyffi ifi u^ining, 

I Q Lacaang addrassas diat did hat ratum 
a quastiannmifa 

a m intan^iawjng paapia (asWn| tha ^yastians) 

^ % Camaleting aia quastiannaira (addrais labal 
'•par Census Usa Only'* bajt, m4 item 2 an 
back pa^a) 

4 Q Campliitini addpasi rtgisw 
(caiymns 10 and 13) 

I FHIing aat payfall farms 

i Uiini jab fafartncf manuals 

7 ^ Gatn^ ptaaj€ m caaparata 

i r^' Daalifii wl^ ynysyai liajatiens 

^ s r*l Answ^mg ayastiant'-asfead by 'respand©it$ 

ia ^ daekinf far missad haysing unis 



1 1 



— Ooiaf - Sptcify 





fallawiBi is a list af diHataat aetiviHas 
in Mlaiflf, 

Atert an fX) in ^e appropriQiB ca/ymn ^at bast 
dasmtas hew aacrt aciiv^/ty ha/ptd yey /tam 
yayf /ab* 



a^ Uaemifa by miner 
(raading fram tfainini 
lyida) 

b. Pficdee inwviawini 
(aidiaf wldi ^inar ar 
fallaw anymeratafs) * < 



a^ Warkteak e^erelsas 



Ra^ini and usin| 
manuals 



9« Audiavisuili (slldas and 
^pa rtMfdin|s) 

I. O^ar - Specify, 



h«ipfyi 


Haip^l 


Least 
helpful 


1 


1 


% 


t 




3 


i 


1 


3 


! 


2 


3 


V 


2 ^ 


3 


1 


2 


3 



sa hff 

Mark (X) ana 

' 1 Q Vary gaad 
2 ^ Gaad 
r Pair 
4 ^ Pear 



7. !f askad» wauid yau work In anathar atnsui iab? 
Mark (X) ana 

, ipYaa ■ 
ipNa^ ^ 

« ^MMidi ofi !ha lab 



i 



8« On tk9 9wfaf## h©^ many HmQt tadi day 



b« QuoitlMMif® reftr^naa took? 



9m Qv#f9ii^ wduld rate Hf9 qyoiify rh^ 
troiiitnf ymo r&€tiv#d f^r this jab? 

1 p Flif 
4 Poor 



Aat ymu ha¥© b##fi elraid wmk In? 
1 □ Noi 

Q Til » What thoyld tlie Bart^y lat y^y 
§b@yt it? Msrk all that QWpiy 

I p Noit^jng spKiai, jyii your norma! job 

3 Q Luvi ^oss iddreiiai is last 

s Q Ask inotiief ffiumtratsr |g |g ^ert 

fi Q Net |Q taek If pwpla are ngt home 
flfst time 

f Q TtH y@uf craw iMdif wark widi yoy 
8 Q Go ^ift Ofiiy Ajfinf tft€ daytime 
0 ri 0*sf - Spigjfy 



10« Tlie followfiig li a llit of %uq%Q%im4 impr^v^merits. 

which you think the SyfMu of ^e Censui CQuld 
mskm to improvm your job 



€tp Improve tfie irajntng • * 

km Pay more money « • ^ . . 

e# Simplify the job (lOt rid 
of manyals) ^ . , • . 

d. Ailow work on Sunday . 

#« Allow Qverdme work , . 

f. Pay by tfie houf, not 
^e ^uMti^niire 



• I • 9 • 



f* Do ill Inttfviewmg ysini 
Ae celephQne . ^ # » . . • » 

h* Have ifiumermtQfS w*3fk 
in 'eams * , . * 

I. Simolify ^e 
qyeitionnairt 

j. Have betier 
syotrvitorf 

km Pay enymtrners on 



I. Othef - S^e€ify 



1 



i 

- 1 
ii 


1 

m m m 


1 

i4 




2 


3 






3 




Z 


3 




z 


3 




2 


3 




2 


3' 




2 


3 


! 


2 


3 


1 


1 


3 


! 


2 


3 


1 


2 


3 


1 


I 


3 



Mm Whof wOl /oyr tmpioym^nt sfafyi whon yoy wor^ 
hired fer the 1?S0 ftfitys? 

^rk (X) one 

1 Q Lookini for a full or part*dme job 

(tempomry or permanent) 

2 □ Not workini, not retiftd, not a stydent, 

not lookini for a job 

3 Q Rttired 

4 Q Swdent 

s p Employed, job %mrm In ^e funjre or 
tempofarily laid off 

i p impioytd 

t p Some*in| alio - Speeif)^ 



13* Commenti 



APPENDIX C JOB PERFORMANCE RECORD 



F OHM D*a57(AT) 



Forni D-275 = Recofd of Tfoining 



U.S. DEPARTMiNT OF COMMLRCE 



ALTERNATIVE TRAINING EXPERIMENT 

FOLLOWUP I ENUMERATOR 
PERFORMANCE RECORD 

20th Decennial Census ^ 1980 



1. D,0, code 



2, C.L,D, No. 



3* Trninififi dnius 
Apr i 1 ^ _io Apr [ ! 



4. Enuinerntor nnmv 



6* ED No. (0000, if reserve) 7. Reviyw test score 



5. Tcleplione f^o. 



8* Did [fu* eriumerniof complete trciininf,' Mark (X) one 
i '^^l Ycis 2 No 3 Dofi't know 



^ Foffns D^26fA or D^267B ~ Field Employee 
Selectton Aid {Enter "99" if no test score) 



r orm D.170 _ QC Enumerotor Doily Progreis Renord 

(Use first two forms for each crew ieoder d is triCL) 



9. S 



eiecciofi test score (itern 



Form D*185 ^ Record of First Review 
(Enter "99'' if forrr) is riQl o vti t ; U b/' e J 



10* N umber of "Nd*s** jh Part A 



11, Par. A re^uhs 

) I Sn t i s ^ncior > 

: ~ Needt> i rrtpr o ven'^ri i 

3 Un snit sf nc tory 

4 No foff' corripieted 



12i NiHr bef of s^iort form errors 



13* Number 'or! I", forr^ errors 



1 4 . Part B re^^ui t 
Mark (Xj o^'p 

.1 ^ ^ 0^4 

2 ^ • 5-20 

1 " ' More thnr^j 20 

4 No fQrrr; Completed 



ForrT>*D^158 Record Qf ReintefViow 

(Enter "99" if person's nanie ib ntil on fonnj 



15, Number of fails! (Look for more than \ 
one box if there^ are any fails J | 



16. Date ( iteni 6 ) 
April 



1 8, Date f Item 6 ) 
April 



17. No. of ncceptnbl^ questionnaires 



19. No. of ncceptnble quustionnaires 



F 



Form D-152 " Crew Ueoder Record of Assignments 

fUse at end of Fo/lowuplj 



20. Did enuiTicrntor complete ED^ ("X" in front of ED No.) 

tAark (X) one 

1 ™ Yes 
^ '^^ No 

3 _ Don't kriow 

4 ^ Not on form 



Form D-291 ^ Employee Poy Vouchor 

(Chock a* er)d of F oi lowup Li 



2 1 * R e s s 0 ' ' for s e p n r a 1 1 0 ft 
Mark (Xi one 

1 _ j AbSifjunent conipleted 

2 Q i Pay dissntisfaciion 

3 |7_ j Work dissatisfaction 

4 [7 i To take another job 

5 LZJ Poor performance 
fe I No fonn nviiilnblt' 
f \ Otfier - Sppc/fy ^ 



^ Forrfi D-153 ^ Crew Leoder Record of Progreis 

fUse fjf end of f fflluwupZ) (Enter '[999" if person 
IS not on form,} 



22. Copy the tlurd pntry for each eiu/tiieralo* 
or Inst entry i 
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APPENDIX D 



V/.1 



APPENDIX D 

TABLE 1 

Comparison of District Offices On Matching Variables 
Based on 1970 Statistics 





Pai 


ir 1 


Pair 2 




Pair 3 






IDA 
J PA 


r 
L 


1 DA 




U 1 n 


f 


Total 














Population 


626106 


622603 


704391 


689104 


644456 


651737 


% Black 


3.1 


5.2 


3.3 


2.8 


2.0 


2.7 


% Urban 


88.3 


98.2 


68.4 


80.9 


100 


100 


Median Age 


"^1 ft 




ps? Q 

^ -J m y 


26 7 


38.3 


33. 0 


Median 














Education 


12.1 


12.2 


12.1 


* 


11.2 


11.4 


Median 














Income 


10290 


10606 


10942 


9856 


11168 


11040 


Allocation i/Pop 


1.0 


1.1 


1.1 


1.1 


1.5 


1.6 


Rates Housing 


2.2 


2.1 


1.8 


2.1 


2.4 


2.5 


Mail-back 














rate 1970 


* 


* 


■ 77.9 


78.0 




* 



* - data not avanable JPA - job perforfnance aided C - Control 

i/ These rates are an estimate of missing census data for population (POP) and 
housing questions* 
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TABLE 2A 

Ratings of Degree of Preparedness Based on Prior Census Experience 
Question Pre: "How well prepared do you feel to go out and work as an enumerator?" 
Post: "How well prepared were you to go out and work as an enumerator?" 

End-of-Tra1n1ng 







Prlop Exptriance 






No Prior Experience 






(145) 


(188) 


(333) 


(1013) 


(895) 


(1908) 




J PA 


Control 


Overai 1 


JPA 


Control 


Overai 1 




a. 


b. 


c. 


d. 


e. 


f. 


Well prepared 


60.0% 


53.2% 


56.21 


46.1% 


47.2% 


46.6% 


Adequately prepared 


40.0 


46.3 


43.5 


53.2 


,50.9 


52.1 


Poorly prepared 


0.0 


0.5 


0.3 


0.7 


1.8 


1.2 


Not prepared 


0.0 


0.0 


0.0 ' 


0.0 

• 


0.1 


0.1 








Post-Training 












Pr1 or Experi ence 






No Prior Experience 






(106) 


(105) 


(211) 


(813) 


(452) 


(1265) 




JPA 


Control 


Overan / 


JPA 


Control 


Overai 1 


Well prepared 


66.01 


58,1« 


62. 1% 


60.1% 


54.0% 


57.9% 


Adequately prepared 


32.1 


37.1 


34.6 


38.7 


44.0 


40.6 


Poorly prepared 


1.9 


3.8 


2.8 


0.9 


2. 0 


1.3 


Not prepared 


0.0 


1.0 


0.5 : 


0.2 


0.0 


0.2 



f^EBsiCNOTE: Base for percentages is reported in parentheses above group desfgnation. 3.I 



N . - 

TABLE 2F? 

Chi-Square Statlstic's for JPA and Control Groups Controning for Prior Census Experfence 

End"Of-Training Post-Training 

2 2 
X p . ^ p 

JPA (Cols. a. vs dj 10,4 .01 2,9 .41 

Control (Cols* vs e,) 3,6 . .31 ' 6,7 .31 

Overall (Cols* vs fO 11.9 .01 6,1 ,11 

NOTE: All comparisons are based on 3 degrees of freedom. 



TABLE 2C . 

Chi-Square Statistics for JPA vs Control Groups for End=of -Training and Post-Training 

Prior Experience No Prior Experience 

2 2 

X p X p 

End-of'-Training 2*2 ,33 . 6,5 ,09 

Post-Training 2,6 . ,45 7.9 ,05 



NOTE: All comparisons are based on 3 degrees of freedom. 



TABLE 3A 

Ratings of Quality of Traininci Based on Prior Census Experience 
Question: "OvGrall. how would you rate the quality of the training?" 



E^n d - 0 f - T r a i n i n g 



Very Good 
Good _ 
Fair 
Poor 



Prior Experience 

(145) (188) 

JPA Control 

a. b. 

62,8% 47.9% 

32.4 44.7 

4.8 7.4 

0.0 0. 0 



(333) 
Overal 1 
c, 

54,4% 

39.3 

6.3 

0.0 



No Prior Experience 



(1018) 


(903) 


JPA 


Control 


d. 


e. 


56.2% 


49.3% 


38.8 


41.5 


4.4 


8.5 


0.8 


0.7 



(1921) 
Overal 1 
f. 

52.9% 

40,0 

6.4 

0.7 



Post-Training 



Very Good 
Good 
Fair 
Poor 



Prior Experience 

(98) (100) (198) 

JPA Control Overal 1 

69.4% 45.0% ■ 57.1% 

25.5 41,0 33.3 

3.1 13.0 8.1 

2.0 1.0 1.5 



NOTE: 



Base for percentages is reported in parentheses above group designation. 



No Prior Experience 



(784) 
JPA 

57.4% 

38.5 

3.1 

1.0 



(426) 
Control 

44.8% 

47.9 

6.8 

0.5 



(1210) 
Overal 1 

53.0% 

41,8 

4.4 

0.8 



*J ^ 



Table 3B 

Chi-Square Statistics for JPA and Control Groups Controning for Prior Census Experienci 

End-of-Training ' Post-Training 

2 2 

X p . X p 

JPA (Cols. a. with d.) 3.4 .33 6.9 .07 

Control (Cols. b. with e.) „ 1.9 .59 5.1 .16 

Overall , (Cols. c. with f . ) 2.6 .46 • ., . 42 .03 



NOTE: All comparisons are based on 3 degrees of freedom 

t 

TABLE 3C 

■Chi-Square Statistics for JPA vs Control Group for End-of-Traini ng and Post-Training 

Prior Experience No Prior Experience 

2 2 
X , p X p 

End-of-Trainlng 7.4 .03 18.1 • # .00 

Post-Training 15.1 .00 23.9 .00 

NOTE- All comparisons are basid on 3 degrees of freedom. 



TABLE 6A ■ 

Percentage of Enumerators Checking Job Tasks Prior to And After Job EKperlence 



* I 

* End-of^Tralning question: "The following Is a list of diyferent pacts of your Job. Mark an (X) by e^ch that you 

think will cause some probl ems when you start working." ^ 

'Post-Training Question: "The following is a list of different parts of your Job. Mark an (X) by each that wjif^ 

not covered well enough In traini ng. *' ^ 



End-^of-Trai nlng 







Prior Experience 






No Prior Expei 


rlence 




J PA 

% 

CI • 


fl89) 
Control 

K 


(335) 
Overall 


(1021) 
J PA 

H 


Control 


fl926) 

Overal 1 
f 


Locating addresses 


7.5% 


8.5% 


8.1% 


8.0% 


6.6% 


7.4% 


Interviewing people 


16.4 


,11.1 


13.4 


11.0 


8.7 


9.9 


wUiM^ ItfUiii^ Uric ^U€^UIUiIlluli€ 


6 S 




5 4 


7 8 


5 9 


6 9' 

- * - 


Completing the address 

ranistpr 


2^7 


4.2 


3.6 


4.9 


6.4 


5,6 


Filling out payroll forms 


4.8 


12.7 


9.3 


8.7 


15.5 


11.9 


Using job reference manuals 


7,5 


13.2 


10.7 


4,8 


8.4 


6.5 


Getting people to cooperate 


4b. 2 


45.5 


45.4 


SO.O 


35,0 


43.0 


Dealing with unusual 
situations 


37.7 


38.1 


37.9 


43.4 


39.3 


41.5 


Answering respondent 
questions 


8,2 


10.1 


, S.3 


10.8 


10.9 


10.9 


Checking, for missed housing 
, units 


11.6 


15.9 


14.0 


20.6 


19.2 


19.9 


NOTE: Base for percentages is 


repor 


ted in parentheses 


above each column. 
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TABLE 6A (Continued) 

Post-Tra1n1ng 

Prior Experience No Prior Experience 





} 

JPA 
a. 


(106) 
Control 
b. 


(213) 
Overall 

c. 


(820) 
JPA 
d. 


(454) 
Control 
e. 


(1274) 
OveraM 
f. 


Locating' addresses 
Interviewing people 


5.6% 
2.8 


5.7% 
0.9 


5.6« 

A 

1/9- 


8.71 
7.4 


10.4% 
8.1 


9.3% 
7.7 


Completing the questionnaire 


2.8 


5.7 


4.2 ■ 


5.5 


9.7 


7.0 


Completing the address 
register 


3.7 


6.6 


5.2 


4.8 


9.7 


6.5 


Filling out payroll forms 


1.9 


10.4 


6.1 


12.6 


17.0 


14.1 


wsing joD reterence manuals 


1,3 


7. 5 


4. 7 


4.8 


8.8 


6.2 


Setting peopVe to cooperate 


15.0 


17.9 


16.4 


14.8 


21 4 


17 1 


Dealing with unusual 
situations 


12.1 


35.8 


23.9 


18.7 


26.2 


21.4 


Answering respondent 
questions 


2.8 


. 5.7 


4.2 


7.4 


9,0 


8.0 


Checking for missed hpusing 
units 


1.9 


14.2 


8.6 


14.1 


15.9 


14,8 



NOTE: Base for percentages Is 



reported in parentheses above each column. 



TABLE SB 

Chl-Squari Statistics For JPA and Control Groups Controning.for Prior Experlenci 

End-of-Training 

JPA Control Overall 



Locating addresses 0.0 

Interylawlng people 3,2 

Completing the questionnaire 0.0 

Completing the address 0.9 
register 

Filling out payroll forms 2.1 

Using job reference manuals 1.4 

Getting people to cooperate 1.0 

Dealing with unusual 1.5 
situations 

Answering respondent 0.6 
questions 

Checking for missed 5.9 
housing units 



0.96 
0.07 
0.80 
0.34 

0,15 
0.23 
0.31 
0.?2 

0.42 

0.01* 



0.6 
0.8 
0,5 

0.9 

0.7 
3.8 
6.9 
0.1 

0,1 

0.9 



0.46 
0,37 
0,48 
0.33 

0.40 
0,05 
0,01* 
0.81 

0.82 

0,33 



0.1 
3.4 
0.8 
1.9 

1.7 
7.2 
0.6 

-^iT4- 

0.6 
6.1 



0.74 
0.06 
0.36 
0.16 

0.19 

0.01* 

0.45 

o:2V 

0.44 
0.01* 



♦Indicates significant difference, p < .05 



TABLE 68 (CONTINUED) 



Post-Training 



JPA 



Control 



Overan 



X 

Locating addresses 0.8 

Interviewing people 2.5 

Cornpletlnq the questionnaire 0.9 

Cortpletlng the address 0,1 
'register 

Filling out payroll forms ,, 9.7 

Using Jcb reference 1.2 
mahuals 

Getting people to cooperate OrO~ 

Dealing with unusual 2.3 
situations 

Answering respondent 2.5 
questions 

Checking for missed 11.8 
housing units 



0.37 
0.12 
0.34 
0.82 

0.00* 
0.26 

IToo^ 

0.13 
0.12 
0.00* 



1.7 
6.0 
ll.3 
0.7 

2.3 
0.1 

0.4 
3.5 

0.9 

0.1 



]0.19 
0.01* 
0.26 
0.42 

0.13 
0.82 

0.51 

0.06 

0.35 
0.77 



2.6 
8.8 
1.8 
0.4 

9.7 

0.5 

0.0 
0.6 

3.2 

6.5 



0.11 
0.00* 
0.18 
0. 55 

0.00* 
0.48 

0.88 
0.45 

0.07 

0.01* 



♦Indicates sign1f(^nt differenee, p < .05 



TABLE 6C 

Chi-Square Statistics for JPA vs Control Groups Controning for Prior Census Experience 
^ End-of-Tralnlng ^ Post-Training 



Prior Experience No Prior Experience Prior Experience No Prior Experience 



\ 


2 

X 


P 


2 

X 


P 


2 

V ^ 
A 


P 


2 

V 


P 


Locating addresses 


0.1 


0.91 


1.2 


0.28 


0.0 


1.00 


0.8 


0,37 


Interviewing people 


1.6 • 


0.21 


2.5 


0.12 


0.2 


0.60 


0.1 


0.73__ 


Completing the 
.questionnaire 


0.7 


0,42 


2.6 


0.11 


0.5 


0.49 


7.3 . 


O.Ol* 


CDmpleting the address 
register 


0.2 


0.67* 


1.8 


0,18 


0.4 


0.53 


.10.9 


0.00* 


FilTing out payroll 
forms 


5.2 


0.02* 


20.2 


0.00* 


5.3 


0.02* 


4,3 


0.04* 


Using Job reference 
manuals' 


2.2 


0.14 


9. 7 


0.00* 


2.7 


0.10 


7.6 


0.01* 


Getting people to 
cooperate 


0.0 


1.00 


43.6 


0.00* 


0.2 


0.69 


8.5 


0,00* 


Dealing with unusual 
situations 


0.0 


1.00 




0. 08 • 


15.1 • 


0.00* ' 


9.5 


0,00* 


Answering respondent 
questions 


0.1 


0.70 


0.0 


0.97 


■ 0.5 


0.49 


0,8 


0.37 


Checking for missed 
housing unfts 


0.9 


0.34 


0.5 


0.50 


9.3 


0.00 


0,6 


0.46 



♦Indicates significant difference, p < •05, 

ERIC 



